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ABSTRACT 


There  is  a  difficulty  in  incorporating  information  value  judgments  into  con¬ 
figuration  management  decisions  regarding  command  and  control  systems. 
This  work  reviews  two  command  and  control  process  models,  decision  theory 
as  it  relates  to  command  and  control  and  the  current  tactical  data  link  config¬ 
uration  management  method.  The  Delphi  method  is  discussed  and  a  means 
of  incorporating  its  use  into  configuration  management  is  introduced.  The 
Delphi  method  allows  a  systematic  gathering  of  subjective  information  from 
selected  respondents  which  then  enables  formulation  of  a  group  position.  Use 
of  this  method  would  enable  subjective  assessments,  such  as  perceived  oper¬ 
ational  impact  of  tactical  data  link  changes,  to  be  systematically  considered  in 
Navy  tactical  data  link  configuration  management  decisions. 
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I.  INTRODUCTION 

Difficulty  exists  when  system  engineers  or  evaluators  try  to  determine  the 
usefulness  of  information  to  a  corporate  or  military  decisionmaker.  As  a  re¬ 
sult.  many  systems  are  designed  without  evaluating  the  decisionmaker's  value 
judgments  on  the  information  being  presented  to  him.  This  is  especially  true 
in  the  area  of  military  command  and  control  systems.  Communications  path¬ 
ways  and  connectivity  issues  are  easier  to  determine  than  the  specific  infor¬ 
mation  needs  of  the  commander.  This  study  will  present  a  method  to  improve 
the  capability  of  incorporating  information  value  into  Navy  configuration  man¬ 
agement  decisions. 

A.  BACKGROUND 

Current  emphasis  in  military  budgeting  gives  rise  to  measures  of  effec¬ 
tiveness  equaling  a  steady  budget  line.  Unfortunately,  this  view  often  leaves 
the  command  and  control  aspect  of  command,  control  and  communications 
unfunded.  Since  it  is  difficult  to  quantify  the  information  needs  of  the 
commander  and  therefore  a  value  cannot  be  placed  on  the  absence  of  the  in¬ 
formation.  many  valid  programs  fail  to  get  the  continued  funding  they  need. 
This  is  exemplified  by  the  delayed  fielding  of  the  Joint  Tactical  Information 
Distribution  System  (JTIDS)  hardware  which  implements  the  TADIL  J  message 
standard.  Originally  to  be  fielded  in  the  late  1970's.  the  Navy  does  not  antic¬ 
ipate  full  implementation  prior  to  1993. 

Within  the  Navy,  some  attempts  have  been  made  to  determine  the  impact 
of  message  translation  between  two  tactical  digital  data  system  standards. 
TADIL  A  (Link  11)  and  TADIL  J  (Link  16).  These  efforts,  although  largely  un¬ 
successful.  have  underscored  the  need  to  incorporate  decisionmakers'  value 
judgments  into  Navy  configuration  management  of  these  data  links.  This  the¬ 
sis  is  an  effort  to  meet  that  need. 
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B.  OBJECTIVE  OF  THE  STUDY 


The  objective  of  this  study  is  to  identify  a  method  to  incorporate  various 
subjective  considerations  that  should  be  taken  into  account  when  making 
configuration  management  decisions  that  affect  the  inclusion  or  exclusion  of 
information  in  a  tactical  digital  data  link.  It  was  suggested  as  a  topic  by  staff 
members  of  the  Navy  Tactical  Interoperability  Support  Activity  (NTISA)  which 
is  chartered  by  the  Chief  of  Naval  Operations  to  do  configuration  management 
(CM)  of  the  data  links  under  the  Joint  Interoperability  of  Tactical  Command 
and  Control  Systems  (JINTACCS)  program. 

C.  SCOPE  OF  THE  STUDY 

The  scope  of  this  study  includes  an  evaluation  of  the  command  and  control 
process  from  the  view  of  the  human  decisionmaker.  This  view  draws  from  both 
classical  and  recent  research  on  decision  theory.  Three  C2  process  models, 
each  of  which  expands  upon  the  other,  will  be  reviewed. 

It  is  a  difficult  to  apply  quantitative  scales  to  information  value  as  per¬ 
ceived  by  the  decisionmaker.  A  method  will  be  introduced  which  may  allow 
for  CM  decisions  to  be  made  in  a  more  scientific  manner, 

D.  LIMITATIONS 

A  thorough  evaluation  of  the  existing  method  used  for  CM  of  the  Navy's 
tactical  data  links  will  not  be  done,  but  sufficient  explanation  of  the  method  will 
be  given  to  help  the  reader  to  understand  the  development  of  the  CM  issues 
discussed.  To  allow  for  wider  distribution  of  this  work  and  to  keep  it  unclas¬ 
sified,  no  pending  configuration  management  items  will  be  discussed.  Exam¬ 
ples  used  are  of  the  author's  creation  and  do  not  represent  known  CM  issues. 
The  focus  of  this  work  is  on  CM  within  the  U.  S.  Navy  although  many  of  the 
concepts  may  apply  equally  well  to  the  other  military  services. 

E.  ASSUMPTIONS 

The  author's  basic  assumption  is  the  Navy  does  not  adequately  include  the 
information  value  judgments  of  the  decisionmaker  in  the  design  and  subse¬ 
quent  configuration  management  of  tactical  data  systems.  The  classical  sci- 


entitle  metiiod  is  applicable  when  discussing  the  process  of  decisionmaking 
but  does  not  lend  itself  to  assigning  value  judgments  to  a  particular  piece  of 
information.  This  inability  to  determine  "value  within  the  Navy's  configura¬ 
tion  management  has  led  to  difficulty  when  determinations  have  to  be  made 
about  whether  to  include  a  particular  information  message  into  the  data  link. 
As  a  result,  members  of  configuration  management  bodies  have  used  various 
heuristics  to  guide  their  decisions.  These  heuristics,  such  as  perceived  oper¬ 
ational  impact,  vary  from  individual  to  individual.  The  group  then  spends 
much  of  its  time  trying  to  determine  the  individual  heuristics  and  compile  them 
into  a  group  position.  This  process  is  fraught  with  difficulty  as  the  military  rank 
of  the  members  tends  to  unduly  influence  the  group  position.  (Bennett.  1989) 
Another  method  must  exist  which  allows  for  heuristics  to  be  taken  into  ac¬ 
count  but  eliminates  the  rank  structuring  of  existing  configuration  management 
decision  making  process.  This  method  should  allow  for  some  anonymity  by 
the  CM  decisionmakers,  allow  for  a  methodical  review  of  issues  and  be  able 
to  treat  non-quantitative  problems.  This  method  should  also  be  easy  to  incor¬ 
porate  into  the  existing  configuration  management  process. 

F.  DEFINITIONS 

The  following  definitions  are  given  to  assist  the  reader.  If  no  reference  is 
given,  the  definition  is  that  of  the  author. 

•  Configuration  management.  A  discipline  applying  technical  and  admin¬ 
istrative  direction  and  surveillance  to; 

1.  Identify  and  document  the  functional  and  physical  characteristics  of  a 
configuration  item. 

2.  Control  changes  to  those  characteristics. 

3.  Record  and  report  change  processing  and  implementation  status 
(NTISA.  1987.  2-2). 

•  Command  and  Control.  The  exercise  of  authority  and  direction  by  a 
properly  designated  commander  over  assigned  forces  in  the  accomplish¬ 
ment  of  the  mission.  Command  ana  control  functions  are  performed 
through  an  arrangement  of  personnel,  equipment,  communications,  facili¬ 
ties.  and  procedures  which  are  employed  by  a  commander  in  planning, 
directing,  coordinating,  and  controlling  forces  and  operations  in  the  ac¬ 
complishment  of  the  mission  (JCS  Pub  1). 


•  Configuration  item.  A  system  and  its  associated  standards  which  need  to 
be  controlled  to  allow  for  optimal  performance  of  the  system. 

G.  ORGANIZATiON  OF  THE  STUDY  ♦ 

Chapter  Two  focuses  on  existing  research  in  the  area  of  command  and 
control  process,  evaluating  two  current  process  models.  Classical  decision  ‘ 

theory  is  discussed  and  related  to  military  decisionmaking.  It  examines  the 
current  configuration  management  process  and  relates  this  to  the  hardware 
and  software  view  of  C3  concluding  with  the  author's  perspective  of  the  human 
role  in  C2. 

Chapter  Three  discusses  the  Delphi  method  and  relates  it  to  use  in  Navy 
configuration  management  with  Chapter  Four  focusing  on  the  implications  of 
the  method  being  used  in  CM.  The  final  chapter  presents  the  author's  con¬ 
clusions  and  recommendations  and  gives  suggestions  for  further  study. 
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II.  VIEWPOINTS  ON  COMMAND  AND  CONTROL 


This  chapter  discusses  various  viewpoints  that  influence  the  focus  of  com¬ 
mand  and  control.  Pertinent  research  on  decision  theory  will  be  presented. 
Two  descriptive  C2  process  models  are  discussed  and  contrasted.  The  chap¬ 
ter  concludes  with  a  description  and  evaluation  of  the  current  U.  S.  Navy 
configuration  management  scheme  for  tactical  digital  data  links. 

A.  WHAT  DOES  DECISION  THEORY  SAY? 

The  modern  roots  of  decision  theory  (DT)  can  be  traced  to  von  Neumann 
and  Morganstern.  an  economist  and  mathematician  respectively,  who  de¬ 
scribed  utility  theory  as  a  method  for  determining  why  people  make  the  deci¬ 
sions  they  do.  A  decision  analyst  uses  the  six  concepts  listed  below  to 
evaluate  the  decisions  made  by  a  particular  decisionmaker.  This  evaluation 
is  used  to  assist  a  decisionmaker  in  acting  in  consonance  with  his  established 
priorities.  These  six  major  concepts  are: 

•  probability 

•  utility 

•  aggregation 

•  decision 

•  choice 

•  preference. 

Probability  and  utility  are  subjective  assignments  unique  to  the  decisionmaker 
and  the  situation.  Probability,  in  this  usage,  is  the  decisionmaker's  determi¬ 
nation  of  the  likelihood  of  occurrence  of  the  situation.  The  utility  of  an  alter¬ 
native  under  consideration  is  in  the  mind  of  the  decisionmaker.  Decision 
theory  assumes  it  must  be  measurable,  at  least  on  an  ordinal,  if  not  a  cardinal 
scale.  This  measurement  would  be  conducted  by  a  decision  analyst  but  is  not 
explicitly  done  by  the  decisionmaker.  The  third  concept  is  aggregation;  the 
decisionmaker's  activity  which  combines  the  probability  and  utility  of  various 
alternatives  in  order  to  choose  between  them.  Decision  theorists  group  deci¬ 
sion,  choice  and  preference  together.  An  analysis  of  the  probability,  utility  and 
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aggregation  values  of  the  decisionmaker  should  lead  to  a  preference  among 
alternatives  with  a  choice  option  between  alternatives.  This  process  would 
then  result  in  a  decision  being  made. 

Decision  theory  researchers  do  not  appear  to  be  bothered  by 
decisionmakers  not  always  acting  in  consonance  with  the  judgments,  such  as 
probability  and  utility,  which  they  assign  to  the  alternatives  under  consider¬ 
ation.  The  researchers  merely  want  to  identify  the  logic  behind  decisions  and 
help  the  decisionmakers  identify  when  they  are  not  acting  in  accordance  with 
their  values.  (Hammond,  1980,  44-67) 

In  its  recent  development,  Keeney  and  Raiffa  .  .  .  emphasize  the  point  that  the  aim 
of  DT  is  to  elaborate  the  logical  entailments  of  the  subjective  probability  and 
utility  theory  and  extend  them  to  a  variety  of  circumstances.  .  .  .  DT  makes  no 
claim  that  it  represents  or  describes  the  cognitive  activity  (or  information  proc¬ 
essing)  of  human  decision  makers.  (Hammond,  1980,  43-44) 

Decision  theorists  avoid  determinations  of  what  decisionmakers  do  but  fo¬ 
cus  instead  on  the  values  and  judgments  which  they  place  on  the  information 
used  to  make  the  decision.  Extending  this  to  the  configuration  management 
process,  one  may  conclude  various  alternatives  under  consideration  will  have 
varying  levels  of  value  to  the  user  based  on  the  perceived  probability  and 
utility  of  the  alternative  to  the  commander.  Assuming  this  is  true,  such  infor¬ 
mation  value  judgments  should  be  considered  in  the  configuration  manage¬ 
ment  of  command  and  control  systems. 

B.  COMMAND  AND  CONTROL  PROCESS  MODELS 

Various  models  have  been  created  in  an  attempt  to  describe  the  command 
and  control  process.  This  section  discusses  Lawson's  model.  Metersky's  ex¬ 
pansion  of  Lawson's  model,  and  then  evaluates  them  in  terms  of  the  human's 
role  in  the  process. 

1.  Lawson's  view 

In  his  report.  "State  Variables  of  a  Command  Control  System"  Lawson 
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Figure  1.  Lawson's  Command  and  Control  Process  Model 


describes  five  basic  functions  of  a  C2  system. ’  As  illustrated  in  Figure  1  these 
five  functions  are: 

•  sense 

•  process 

•  compare 

•  decide 

•  act. 


i  This  section  is  the  author's  evaluation  of  Lawson,  J.  S.,"The  Stale  Variables  of  a  Com¬ 
mand  Control  System."  Proceedings  for  Quandtalive  Assessment  of  the  Utility  of  Command 
and  Control  Systems.  Office  of  the  Secretary  of  Defense  with  the  cooperation  of  the  MITRE 
Corporation.  C’Division  (Washington,  D.C.:  National  Defense  University,  Ft.  Leslie  J.  McNair, 
January  1980),  93-99. 
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As  described  by  Lawson,  sensing  is  the  gathering  of  signals  in  the  form 
of  data  from  the  environment.  This  data  can  include  radar  returns,  imagery 
reports  or  reports  from  human  observers.  This  data  is  then  processed  and  * 

external  data  may  be  introduced.  Processing  implies  the  conversion  of  raw 

« 

data  to  information  which  can  then  be  used  to  compare  the  existing  state  to  the 
desired  state. 

After  comparing,  the  next  function  performed  is  decide.  The  decide 
function  can  be  supported  by  decision  aids  and  is  based  on  information  from 
the  previous  three  functions.  These  decision  aids  can  include  automated  data 
bases  which  allow  the  decisionmaker  to  review  historical  information  about  the 
situation,  graphical  plots  of  units'  position,  or  expert  systems  which  can  re¬ 
commend  courses  of  action  based  on  the  information  available. 

The  last  function  is  act.  The  goal  of  this  function  is  to  influence  the 
environment  to  produce  the  desired  state  as  used  in  the  compare  function.  In 
many  cases,  actions  are  communicated  to  higher  authorities.  Input  from 
higher  authorities  may  then  enter  the  loop  as  external  data  in  the  process  ' 

function. 

2.  Metersky's  view 

In  1986,  Dr.  Mort  Metersky  of  the  Naval  Air  Development  Center  pub¬ 
lished  a  paper  expanding  Lawson's  C2  process  model, 2  In  this  he  states: 

[t]he  system  is  a  combination  of  elements  that  form  a  complex  whole.  This 
system  is  composed  of  two  inter-dependent  parts:  1)a  C*  process  consisting 
of  people,  procedures. and  organizations  and  2)  physical  components,  e.g.,  sen¬ 
sors.  computers,  and  communications.  The  major  function  of  the  C*  process  is 
to  provide  decision  specific  information  to  the  responsible  decisionmaker, 
whereas  the  physical  components  provide  the  mechanism  for  initiating  and  im¬ 
plementing  the  C‘  process.  (Metersky.  1986,  880)  (italics  added  for  emphasis) 

Metersky  states  there  are  three  perspectives  of  command  and  control 
systems  that  should  each  be  evaluated  with  various  techniques  from  different 

2  This  section  is  the  author's  evaluation  of  Metersky.  M.  L..  "  A  C'  Process  and  an  Ap-  • 

proach  to  Design  and  Evaluation."  IEEE  Transactions  on  Systems.  Man.  and  Cybernetics,  v. 

SMC-16.  no.  6.  November/December  1986:  880-889. 
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disciplines.  These  three  perspectives  are  the  totai  system,  the  C2  process  and 
the  physical  components.  (Metersky,  1986.  880) 

Metersky  takes  each  of  the  five  functions  identified  in  Lawson  s  model 
and  expands  them  to  specifically  include  each  subfunction  necessary  to  do  the 
function  and  the  organizational  relationships  that  are  implied  by  Lawson  s 
treatment. 


Figure  2.  Metersky's  Sense  Expansion 


In  Figure  2  note  the  explicit  representation  of  the  expanded  sense  function. 
As  in  Lawson's  model,  the  sense  function  is  devoted  to  assimilation  of  raw 
data  from  various  sources.  An  additional  data  set  included  is  the  perception 
of  the  enemy's  reaction  to  the  action  taken.  This  assimilated  data  then  serves 
as  input  to  the  process  function. 
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* 

Metersky  groups  the  process  and  compere  functions  since  both  work  together 

to  provide  information  to  the  decisionmaker.  In  both  the  Lawson  and  Metersky  ^ 

models  the  decisionmaker  may  be  either  a  human  or  a  machine.  Noteworthy 

is  Metersky's  view  of  the  decision  requirements  of  the  decisionmaker  within 

the  organizational  structure  being  the  driving  force  behind  the  compare 

method  used.  Figure  3  refers. 

Figure  4  on  page  11  shows  the  expanded  version  of  the  decide  func¬ 
tion.  Within  the  dotted  box,  he  includes  various  cognitive  factors  that  affect 
decisionmaking  by  a  human.  These  factors  include  the  decisionmaker's  de¬ 
termination  of  the  quality  of  the  input  received,  human  filtering  such  as  expe¬ 
rience,  both  recent  and  past  history,  biases  and  effects  of  the  environment. 

The  decisionmaker  takes  all  this  information  and  internally  processes  it  to 

come  to  a  decision.  The  cognitive  factors  included  by  Metersky  as  germane  « 

to  the  decide  function  can  also  be  introduced  Into  expert  system  machines  and 
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other  tactical  decision  aids,  such  as  Link  11,  which  can  assist  the 
decisionmaker  in  his  duties. 
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Figure  5.  Metersky's  Act  Expansion 


The  last  expanded  function  is  act  as  shown  in  Figure  5.  The  subfunction  de¬ 
cision  command  is  the  translation  of  a  decision  into  observable  actions.  The 
second  subfunction  implement  decision  .  .  concerns  the  procedures  taken 
to  effect  the  decision"  (Metersky,  1986,  885). 

Figure  6  on  page  13  combines  all  the  functions  and  subfunctions  into 
a  whole.  He  perceives  several  advantages  to  his  model  which  include  the  re¬ 
cognition  of  humans  being  involved  in  the  process  and  identification  of  specific 
information  transfer  requirements  between  the  various  functions  and  subf¬ 
unctions.  Information  transfer  is  specifically  shown  in  various  places  on  Fig¬ 
ure  6.  (Metersky,  1986.  887) 

3.  Where  does  the  human  fft  In? 

Command  and  control  processes,  when  implemented,  can  vary  from 
the  fully  automated  such  as  one  proposal  for  the  Strategic  Defense  Initiative 
to  the  fully  manual  such  as  the  lone  rifleman  stalking  a  target.  Most  imple¬ 
mented  processes  or  systems  fall  somewhere  in  between. 
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The  system  should  support  the  decision  and  action  functions  of  the 
decisionmaker  in  the  partially  automated  C2  system.  To  do  this,  consideration 
should  be  given  to  the  information  value  judgments  of  the  system  user.  This 
can  be  done  through  decision  analysis  methods  as  discussed  earlier  or 
through  the  method  proposed  in  Chapter  Three. 

C.  CURRENT  DATA  LINK  CONFIGURATION  MANAGEMENT 

The  Navy  Joint  Interoperability  of  Tactical  Command  and  Control  Systems 
(JINTACCS)  Configuration  Management  Plan  (NJCMP)  Revision  1  dated  11 
August  1987: 

.  .  .  details  the  initiation,  processing,  analysis,  testing,  approval,  implementation, 
and  status  accounting  procedures  needed  to  control  and  maintain  JINTACCS 
message  standard  baseline  and  operating  procedures  within  the  Navy.  The 
JINTACCS  program  includes  standards  for: 

a.  Joint  Tactical  Air  Operations  (JTAO)  (Link  11,  4A,  and  14). 

b.  U.S.  Message  Text  Formats  (USMTF). 

c.  Tactical  Digital  Information  Link  (TADIL)  J  standards  for  the  Joint  Tactical  In¬ 
formation  Distribution  System  (JTIDS)  (NTISA,  1967,  promulgation  letter). 

This  section  outlines  the  means  by  which  a  hypothetical  change  proposal 
would  be  evaluated  under  the  existing  configuration  management  (CM)  pro¬ 
cedures  as  described  in  the  NJCMP. 

Configuration  management  under  the  JINTACCS  orogram  includes  these 
basic  elements.  Configuration  identification  is  the  specific  documentation 
which  supports  a  particular  configuration  item.  It  consists  of  operational  spec¬ 
ifications.  technical  design  plans  or  related  JCS  documents  and  will  be  main¬ 
tained  throughout  the  lifecycle  of  the  Cl.  Configuration  control  is  comprised 
of  three  parts: 

•  formal  processing  of  applicable  change  proposals 

•  actual  processing  and  evaluation  of  the  change  proposals 

•  implementation  of  approved  change  proposals  into  the  configuration  item. 

These  JINTACCS  CM  elements  are  incorporated  into  Navy  configuration  man¬ 
agement. 

The  Navy  Tactical  Interoperability  Support  Activity  (NTISA)  is  the  CNO  s 
primary  technical  advisor  for  configuration  control  of  the  JINTACCS  baseline. 
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NTISA  performs  this  function  through  the  Operational  Interoperability  Re¬ 
quirements  Group  (OIRG)  and  the  Technical  Interoperability  Standards  Group 
(TISG).  The  former  group  "is  composed  of  representatives  from  the  operating 
forces  (Fleet  CINCs.  type  commanders,  and  Commander.  Naval  Intelligence 
Command  (COMNAVINTCOM))  "(NTISA.  1987.  3-2).  The  latter  group  is  prima¬ 
rily  responsible  for  technical  aspects  of  the  CIs,  including  maintenance  of  the 
various  operational  specifications.  Its  members  include  representatives  from 
the  various  “software  development  and  life-cycle  support  activities  for  the 
various  tactical  data  system  programs"(NTISA,  1987,  3-2). 

1.  A  Configuration  Management  Example 

Lieutenant  Mary  Smith  is  the  Tactical  Action  Officer  aboard  the  USS 
CARMEL,  a  Link  11-capable  destroyer  assigned  to  Second  Fleet.  In  the  per¬ 
formance  of  her  duties,  she  observes  existing  procedures  for  assignment  of 
the  Net  Control  Station  in  the  data  link  do  not  adequately  consider  radio 
equipment  capabilities  of  the  assigned  platform.  She  believes  an  algorithm 
which  considers  equipment  status  of  the  proposed  NCS  would  allow  for  better 
performance  of  the  data  link  within  the  battle  group.  After  creating  an  algo¬ 
rithm  that  considers  equipment  status,  she  drafts  a  change  proposal  recom¬ 
mending  implementation  of  her  algorithm  and  forwards  it  to  Commander. 
Second  Fleet,  who  is  a  primary  command  in  the  OIRG.  via  the  administrative 
chain  of  command.  Second  Fleet  staff  assigned  to  review  the  proposal  believe 
the  proposed  algorithm  is  a  valid  inclusion  for  Link  11  and  forward  the  pro¬ 
posal  to  NTISA  for  status  accounting.  Second  Fleet  assigned  the  change  pro¬ 
posal  a  characteristic  of  Proposed  Operational  Requirement,  since  the  current 
procedure,  although  not  as  effective  as  LT  Smith's  algorithm,  is  functioning 
well  enough  to  meet  their  operational  requirements. 

Upon  receipt  of  the  Proposed  Operational  Requirement  (POR)  from 
Second  Fleet.  NTISA  will  assign  it  an  agenda  item  number  for  the  next 
biannual  meeting  of  the  OIRG.  It  will  also  be  assigned  a  POR  number  which 

.  .  .  will  be  composed  of  the  functional  area  affected  by  the  POR,  followed  by  a 
sequential  number  relative  to  all  PORs  proposed  in  that  functional  area,  followed 
by  the  calendar  year  in  which  the  POR  was  drafted  (not  necessarily  the  year  in 


15 


which  originated  by  the  source.  Example;  ASW-2-79  is  the  second  ASW  POR 
drafted  in  CY-79).  TADIL  functional  areas  are  "AAW".  "ASW",  "ASUW",  "EW", 
or  “OPS"  (NTISA,  1987,  7-3). 

This  POR  number  allows  for  status  accounting  of  the  change  proposal  as  it 
makes  its  way  through  the  configuration  management  process. 

NTISA  collects  all  agenda  items  and  provides  a  complete  package  to 
each  of  the  OIRG  participants  at  least  ten  weeks  prior  to  the  next  scheduled 
OIRG.  Each  participant  reviews  the  agenda  items  and  staffs  them  with  subor¬ 
dinate  commands  as  necessary  to  come  up  with  a  command  position  on  each 
item.  No  procedures  are  in  the  NJCMP  for  the  method  to  be  used  by  the  par¬ 
ticipants  in  doing  their  internal  staffing  of  agenda  items. 

At  the  OIRG  meeting,  each  agenda  item  will  be  considered  in  turn  by 
the  body.  Items  may  be  agreed  upon  by  the  group  with  one  of  five  disp¬ 
ositions; 

1.  Agree  with  the  proposal  as  written. 

2.  Agree  with  the  item  as  amended  by  the  group. 

3.  Disapprove  the  item, 

4.  Continue  the  item  to  the  next  meeting. 

5.  Withdraw  the  item  at  the  request  of  the  originator. 

If  unanimous  agreement  cannot  be  reached  on  the  agenda  item,  the  Chair  can 
rule  directly  on  the  item  and  record  the  dissenting  opinions,  create  a  working 
group  to  attempt  resolution,  continue  the  item  to  the  next  meeting  or  some 
other  specified  date  or  declare  the  issue  to  be  substantive,  which  would  re¬ 
quire  special  procedures  for  resolution.  (NTISA.  1987.  B-4.  5) 

If  the  OIRG  determines  the  POR  is  of  value  and  approves  it.  the  agenda 
item  will  be  passed  to  the  TISG  for  action.  The  TISG  reviews  the  item  on  its 
technical  merit  and  determines  its  Implementation  schedule  and  cost.  They 
use  a  method  similar  to  the  one  used  by  the  OIRG.  After  the  schedule  and  cost 
are  delineated,  the  item  is  forward  to  the  CNO  for  approval.  NTISA  will  con¬ 
tinue  to  monitor  the  progress  of  the  agenda  item  through  its  implementation, 
after  which  the  item  will  become  a  part  of  the  configuration  baseline. 
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D.  PUTTING  IT  ALL  TOGETHER 

The  previous  sections  discuss  the  command  and  control  process  and  imply 
that  information  within  the  process  has  a  certain  value  to  the  tactical 
decisionmaker.  If  this  is  the  case,  one  would  assume  information  value  would 
be  an  integral  part  of  the  Navy  CM  process.  Unfortunately,  you  will  note  the 
configuration  management  process  as  outlined  in  the  NJCMP.  a  hypothetical 
example  of  which  was  given,  does  not  explicitly  include  a  method  for  subjec¬ 
tive  evaluation  of  the  value  of  information  content  in  the  data  link.  The  next 
chapter  describes  an  explicit  method  for  incorporating  these  subjective  views. 
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III.  INCORPORATING  THE  DELPHI  METHOD  INTO  CONFIGURATION 

MANAGEMENT 


This  chapter  discusses  the  Delphi  method  and  gives  the  author's  views  on 
how  it  can  be  incorporated  into  existing  Navy  configuration  management 
processes. 

A.  THE  DELPHI  METHOD 

Delphi  has  come  a  long  way  in  its  brief  history,  and  it  has  a  long  way  to  go.  Since 
its  invention  ...  for  the  purpose  of  estimating  the  probable  effects  of  a  massive 
atomic  bombing  attack  on  the  United  States,  and  its  subsequent  application  in  the 
mid-sixties  to  technological  forecasting,  its  use  has  proliferated  in  the  United 
States  and  abroad.  While  its  principal  area  of  application  has  remained  that  of 
technological  forecasting,  it  has  been  used  in  many  other  contexts  in  which 
judgmental  information  is  indispensable  (Linstone  and  Turoff  (eds.).  1975.  xix). 

1 .  History 

The  Delphi  technique  was  developed  during  the  early  1950's  for  use  in 
Rand  Corporation's  study  of  the  "application  of  ‘expert  opinion  to  the  se¬ 
lection.  from  the  point  of  view  of  a  Soviet  strategic  planner,  of  an  optimal  U. 

S.  industrial  target  system  and  to  the  estimation  of  the  number  of  A-bombs 
required  to  reduce  the  munitions  output  by  a  prescribed  amount.'  "  This  ori¬ 
ginal  study,  sponsored  by  the  Air  Force,  was  titled  "Project  Delphi".  Devel¬ 
oped  during  this  study  specifically  as  a  long-range  forecasting  tool.  Delphi  has 
crept  beyond  military  boundaries  and  is  now  used  extensively  in  the  private 
sector  for  numerous  applications. 

2.  Method  of  use 

Several  different  variants  of  the  Delphi  technique  have  been  estab¬ 
lished.  This  discussion  will  focus  on  what  Linstone  and  Turoff  (1975)  refer  to 
as  the  "Conventional"  approach.  They  view  Delphi  .  .  as  a  method  for 
structuring  a  group  communication  process  so  that  the  process  is  effective  in 
allowing  a  group  of  individuals,  as  a  whole,  to  deal  with  a  complex  problem" 
(Linstone  and  Turoff  (eds.).  1975.  3).  This  process  allows  for  direct  communi¬ 
cation  of  individual  ideas  and  feedback  of  those  ideas  to  other  participants. 
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assimilation  of  a  group  position,  ability  for  an  individual  to  revise  his  or  her 
views  and  some  amount  of  anonymity  among  the  participants.  These  features 
eliminate  some  of  the  negative  aspects  of  group  decisionmaking  such  as 
“  .  .  .conflict,  disagreement,  and  misunderstanding  among  group  members-- 
[which]  are  regarded  as  products  of  the  inability  of  individuals  to  process  in¬ 
formation  consistently  and  to  understand  the  positions  taken  and  judgments 
made  by  other  members  about  decision  issues  "(Guzzo  (ed.).  1982.  9). 

The  Delphi  procedure  is  comprised  of  various  rounds  which  consist  of 
participants  completing  a  questionnaire  about  the  issues  which  are  to  be  de¬ 
cided  upon.  The  participant  is  usually  given  two  copies  o' the  questionnaire 
and  is  asked  to  fill  out  both  but  return  only  one.  After  the  questionnaires  are 
returned  to  the  supervisor  they  are  collated  into  a  group  position.  On  the 
successive  rounds,  this  group  position  is  provided  to  the  participants  in  addi¬ 
tion  to  the  next  questionnaire.  Several  methods  are  available  to  the  supervi¬ 
sor  to  use  in  forming  the  group  position  from  the  individual  questionnaires, 
but  they  are  beyond  the  scope  of  this  study.  At  any  time  during  the  rounds, 
an  individual  with  an  extreme  position  is  asked  to  give  reasons  for  the  answers 
given.  These  answers  may  also  be  provided  to  the  other  participants  during 
later  rounds.  After  the  selected  number  of  rounds  or  when  the  supervisor 
determines  a  group  position  has  been  reached,  the  Delphi  concludes.  Each 
participant  is  provided  a  summary  of  the  final  conclusions. 

3.  Drawbacks  and  Criticisms 

As  outlined,  the  Delphi  appears  fairly  straightforward  in  its  application. 
However,  there  is  a  significant  overhead  burden  to  the  supervisor  and  his 
team  in  collecting  all  the  answers  and  redistributing  them  to  the  participants. 
Also,  the  construction  of  questions  for  the  Delphi  is  difficult  since  the  supervi¬ 
sor  should  not  attempt  to  influence  the  choices  of  the  participants.  The  journal 
Technological  Forecasting  and  Social  Change  gives  many  examples  of  Delphi 
applications  which  show  adequate  consideration  of  the  former. 
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The  method  is  not  without  its  critics,  one  of  the  most  vocal  of  whom  is 
affiliated  with  the  “’fountain  of  knowledge";  the  Rand  Corporation.  Sackman 
(Rand.  1974)  criticized  the  Delphi  on  two  major  fronts,  the  first  of  which  was 

[i]f  Delphi  is  to  be  treated  seriously  as  a  professional  technique,  it  must  be  judged 
by  basic,  minimum  standards  applicable  to  all  empirical  social  sciences  (Rand, 
1974,  11). 

He  refers  primarily  to  the  American  Psychological  Association's  evaluative 
criteria  established  to  inculcate  scientific  procedures  into  a  field  which  had  a 
long  history  of  abuses  on  the  part  of  test  administrators  and  other  researchers 
(Rand,  1974.  9).  His  second  criticism  was  the  questionable  application  of  the 
technique  by  some  researchers  who  disregarded  basic  points  such  as  sample 
size,  questionnaire  construction  and  reliability  of  their  research.  He  continues 
that  evaluative  judgments  made  from  improperly  constructed  Delphi  are  sus¬ 
pect.  (Rand.  1974.  28-68) 

After  Sackman's  critique,  almost  a  complete  issue  of  Technological 
Forecasting  and  Social  Change,  the  leading  journal  for  Delphi  research,  was 
devoted  to  contradicting  arguments  (7.  127-233.  1975).  The  result  of  this  de¬ 
bate  appeared  to  confirm  Sackman  is  correct  on  his  first  point  but  incorrect  on 
the  second.  This  debate,  although  more  than  ten  years  ago.  should  influence 
the  application  of  any  Delphi.  Commentary  on  dissertations  using  the  Delphi 
method  indicate  all  researchers  are  not  as  thorough  as  they  should  be  in  the 
application  of  the  Delphi  (Rieger.  1986.  201). 

B.  HOW  IT  CAN  BE  USED  BY  NTISA 

The  author  proposes  a  modification  of  Turoff's  Policy  Delphi  as  a  prepara¬ 
tion  technique  for  use  in  the  configuration  management  process.  The  tech¬ 
nique's  use.  as  proposed,  will  not  eliminate  the  need  for  the  existing 
face-to-face  meetings  but  is  intended  to  standardize  the  participants'  prepara¬ 
tion  and  focus  on  explicit  consideration  of  the  impact  of  the  proposed  changes 
on  the  human  decisionmaker.  The  goal  of  including  this  method  as  a  prepa¬ 
ration  technique  is  not  for  the  CM  body  to  evaluate  the  method  by  which  they 
reach  a  decision  on  a  particular  configuration  item.  It  is  more  the  subjective 
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assessments  of  the  participants  regarding  the  information  value  of  the  Cl  in 
question.  Turoff  describes  six  steps  in  the  communication  process  of  his  Pol- 
>  icy  Delphi.  They  are; 

1.  Formulation  of  the  issues.  What  is  the  issue  that  really  should  be  under 
consideration?  How  should  it  be  stated? 

2.  Exposing  the  options.  Given  the  issue,  what  are  the  policy  options  avail¬ 
able? 

3.  Determining  initial  position  on  the  issues.  Which  are  the  ones  everyone 
already  agrees  upon  and  which  are  the  unimportant  ones  to  be  dis¬ 
carded?  Which  are  the  ones  exhibiting  disagreement  among  the  re¬ 
spondents? 

4.  Exploring  and  obtaining  the  reasons  for  disagreements.  What  underlying 
assumptions,  views,  or  facts  are  being  used  by  the  individuals  to  support 
their  respective  positions? 

5.  Evaluating  the  underlying  reasons.  How  does  the  group  view  the  separate 
arguments  used  to  defend  various  positions  and  how  do  they  compare  to 
one  another  on  a  relative  basis? 

6.  Reevaluating  the  options.  Reevaluation  is  based  upon  the  views  of  the 
underlying  “evidence"  and  the  assessment  of  its  relevance  to  each  posi¬ 
tion  taken,  (tinstone  and  Turoff  (eds.),  1975,  88) 

Each  of  these  six  points  should  be  kept  in  mind  when  reading  the  following. 

Upon  receipt  of  the  agenda  items  proposed  for  the  next  OIRG  or  TISG 
meeting  NTISA  will  prepare  a  series  of  questions  relating  to  each  agenda 
item's  possible  impact  on  the  data  link.  Possible  questions  are  included  in  a 
1979  report  by  Applied  Decision  Analysis  titled  An  Analytic  Characterization 
of  Navy  Command  and  Control  Decisions.  Example  questions  from  the  Applied 
Decision  Analysis  report  and  the  rating  scales  suggested  by  Turoff  are  in¬ 
cluded  at  the  Appendix.  The  primary  and  alternate  commands  would  receive 
these  questions  with  the  agenda  item  package.  The  questionnaire  is  accom¬ 
panied  by  instructions  such  as  those  of  Jones  (1978,  99-155).  Additionally,  the 
participants  are  instructed  they  may  consult  with  any  member  of  their  subor¬ 
dinate  staff  in  determining  their  answers  but  the  original  participant  command 
,  representative  should  complete  the  questionnaire. 

After  the  initial  round,  the  questionnaires  will  be  returned  to  NTISA  where 
.  they  will  be  compiled.  The  author  recommends  that  opinions  which  are  away 

from  the  norm  be  included  in  the  compiled  product  along  with  the  reasons 
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given  for  the  answer  by  the  participant.  Following  Jones,  the  second  round 
asks  for  further  elicitation  of  the  issues.  The  results  of  the  first  round  are  in¬ 
cluded  with  the  questionnaire  and  other  materials  for  the  second  round.  The 
third  round  would  follow  the  same  general  procedure  with  the  exception  of  the 
results  being  provided  to  the  participants  at  the  scheduled  OIRG  or  TISG 
meeting.  By  this  point,  the  participants  will  be  clear  on  the  issues  involved 
and  also  be  aware  of  conflicting  views. 

The  author  suggests  this  method  not  be  used  directly  for  issues  involving 
technical  design  standards  such  as  how  to  represent  a  proposed  message  in 
octal  coding  but  that  it  be  reserved  for  review  of  issues  which  affect  the  human 
interface  to  the  data  link.  These  issues  could  range  from  the  determination  of 
effects  from  changing  an  existing  message  field  to  discerning  the  operational 
impact  of  a  change  proposal  to  Standard  Operating  Procedures.  The  focus 
should  be  on  improving  the  utility  of  the  data  link  from  the  perspective  of  the 
decisionmaker. 

Turoff  suggests  a  three  or  four  round  Delphi  may  be  sufficient  if  the  su¬ 
pervisory  team  devotes  a  considerable  amount  of  time  formulating  the  ques¬ 
tionnaire  to  include  all  obvious  issues,  allows  participants  to  add  issues  to  the 
initial  list  and  asks  for  a  position,  with  the  participant's  reasons  for  the  posi¬ 
tion.  on  the  first  round,  (tinstone  and  Turoff  (eds.).  1975.  88-92) 


IV.  IMPLICATIONS  IN  USING  THE  DELPHI 


The  considerations  that  enable  good  decisionmaking  remain  the  same  regardless 
if  the  decisionmaker  is  a  corporate  executive  faced  with  a  problem  in  production 
or  a  military  commander  faced  with  a  tactical  decision.  Each  has  a  responsibility 
to  analyze  the  problem  and  make  a  decision  that  will  affect  the  welfare  of  the  or¬ 
ganization  (Combined  Arms  and  Services  Staff  School,  1984.  1). 

As  noted  in  the  quotation  above,  the  military  commander's  decisionmaking 
responsibilities  are  to  analyse  the  problem  and  make  a  decision.  In  Chapter 
Two.  relevant  concepts  of  decision  theory  were  discussed.  One  primary  con¬ 
cept  was  the  inherent  value  of  information  to  the  decisionmaker.  Chapter 
Three  described  a  method  for  incorporating  information  value  in  CM  deci¬ 
sions.  This  chapter  discusses  implications  of  using  the  Delphi  method  to  in¬ 
clude  information  value  judgments  into  the  CM  process. 

A.  BETTER  SUPPORT  FOR  THE  TACTICAL  DECISIONMAKER 

As  discussed  in  the  section  on  command  and  control  processes,  there  is 
often  little  consideration  given  to  the  "man  in  the  loop"  by  command  and 
control  system  designers.  This  often  results  in  systems  which  do  not  meet  the 
needs  of  the  user.  By  including  a  method  explicitly  requiring  consideration 
of  the  human,  the  human  s  information  needs  will  more  likely  be  met.  This 
point  is  emphasized  by  Metersky  and  Ryder  (undated.  2)  who  state  ‘'[f]or  ef¬ 
fective  task  performance,  the  information  displayed  must  be  relevant  to  the 
operator's  needs."  When  the  decisionmaker's  need  for  information  is  met  you 
may  conclude  the  mission  performance  of  the  unit  to  which  the  decisionmaker 
is  assigned  will  improve. 

B.  IMPROVED  CONFIGURATION  MANAGEMENT 

The  existing  CM  process  does  not  allow  for  a  methodical  evaluation  of 
non-quantitative  issues.  As  a  result,  much  time  is  spent  trying  to  determine 
the  heuristics  on  which  individuals  base  their  CM  votes.  Through  selective 
use  of  the  Delphi  the  OIRG  and  TISG  can  spend  more  time  in  preparation  for 
their  respective  meetings  with  less  time  debating  minor  points  and  arguing 


over  such  things  as  meanings  of  words  during  the  meeting.  Use  of  the  Delphi 
should  also  remove  some  of  the  obstacles  inherent  in  group  meetings  as  dis¬ 
cussed  earlier.  Implementation  of  the  Delphi  should  result  in  a  more  rapid 
treatment  of  agenda  items  and  subsequently,  sooner  implementation  into  the 
data  links. 


< 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


The  author  concludes  that  insufficient  emphasis  has  been  placed  on  the 
human's  role  in  the  command  and  control  process.  The  primary  focus  has 
been  on  hardware  and  software  since  those  items  can  be  seen  and  touched. 
As  a  consequence,  little  emphasis  is  put  on  the  information  needs  of  the  tac¬ 
tical  decisionmaker  in  existing  C2  systems  which  are  ostensibly  designed  to 
support  him.  Various  methods,  including  the  Delphi,  have  been  developed  to 
gather  and  evaluate  expert  opinions  on  subjects  which  defy  quantification 
through  existing  statistical  methods.  However,  these  methods  are  not  being 
used  in  the  Navy  CM  process.  Incorporating  methods  such  as  the  Delphi 
would  improve  the  effectiveness  of  a  C2  system  by  considering  the  information 
needs  of  the  human  user  in  its  design.  This  inclusion  should  result  in  in¬ 
creased  effectiveness  of  the  tactical  unit. 

A.  RECOMMENDATIONS 

Consideration  should  be  given  to  modifying  the  existing  CM  procedures  to 
include  the  modified  Policy  Delphi  method  as  outlined  in  Chapter  Three.  Al¬ 
ternatively.  if  the  Delphi  is  not  included,  NTISA  may  wish  to  use  the  analysis 
method  outlined  in  the  paper  by  Applied  Decision  Analysis  (1979)  to  guide  the 
agenda  item  discussions  of  the  OIRG  and  TISG.  Both  methods  would  force  the 
consideration  of  the  human's  information  needs. 

The  author  proposes  other  research  to  create  a  Delphi  for  use  in  the  CM 
process  described  in  this  study.  This  Delphi  would  include  actual  configura¬ 
tion  items  under  consideration  and  build  on  the  techniques  outlined  in  Chapter 
Three.  Subsequent  analysis  could  include  polling  the  participants'  opinions 
on  the  execution  of  the  method  and  comparative  studies  between  Delphi- 
preparation  and  non-Delphi-preparation  agenda  items  regarding  ease  of 
reaching  group  consensus  at  the  OIRG/TISG  meeting. 

A  second  area  for  further  study  would  be  to  create  a  simulation  program 
for  use  by  NTISA  to  evaluate  independently  a  proposed  change  to  the  data 
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link.  This  program  would  require  the  isolation  of  the  proposed  change  as  the 
independent  variable  and  would  require  almost  full  implementation  of  the  data 
link  standard  into  the  simulation.  The  simulation  results  could  be  used  by  the 
OIRG  in  evaluating  operational  impact  of  a  proposed  change  and  the  TISG 
could  use  it  as  a  method  to  determine  technical  feasibility  of  a  change. 


APPENDIX  A.  EXAMPLE  QUESTIONS 


1.  How  much  time  is  available  for  the  decision?  This  is  the  time  from  re¬ 
cognition  of  a  decision  to  the  point  where  an  action  must  be  taken. 

•  Less  than  one  minute 

•  One  minute  to  one  hour 

•  One  hour  to  one  day 

•  One  day  to  one  month 

•  Greater  than  one  month 

2.  What  is  the  frequency  of  the  decision?  What  is  the  mean  time  between 
recurrence  of  the  decision? 

•  Less  than  ten  minutes 

•  Ten  minutes  to  ten  hours 

•  Ten  hours  to  one  month 

•  One  month  to  one  year 

•  Greater  than  one  year 

3.  What  is  the  quantity  of  information  relating  to  a  decision  received  by  a 
decisionmaker  and  his  staff,  in  terms  of  the  number  of  bytes  per  day  (or 
a  Shannon  information  measure  per  unit  time)? 

•  Less  than  ten 

•  Ten  to  100 

•  Greater  than  100 

4.  What  percentage  of  messages  is  significantly  more  valuable  than  the  av¬ 
erage? 

•  Zero  to  one  percent 

•  One  to  ten  percent 

•  More  than  ten  percent 

5.  How  reliable  are  the  information  sources?  What  percentage  of  the  infor¬ 
mation  sources  are  considered  reliable? 

•  Less  than  forty  percent 

•  Forty  to  sixty  percent 

•  Sixty  to  eighty  percent 

•  More  than  eighty  percent 
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APPENDIX  B.  TUROFF'S  POLICY  DELPHI  SCALES  v 


Turoff's  Policy  Delphi  Scales 


IMPORTANCE  (PRIORITY  OR  RELE¬ 
VANCE) 

DESCRIPTIVE  GUIDELINES 

Very  important 

•  A  most  relevant  point 

•  First-order  priority 

•  Has  direct  bearing  on  major  is¬ 
sues 

•  Must  be  resolved,  dealt  with,  or 
treated 

Important 

•  Is  relevant  to  the  issue 

•  Second-order  priority 

•  Significant  impact  but  not  until 
other  items  are  treated 

•  Does  not  have  to  be  fully  resolved 

Slightly  important 

•  Insignificant 

•  Third-order  priority 

•  Has  little  importance 

•  Not  a  determining  factor  to  major 
issue 

Unimportant 

•  No  relevance 

•  No  priority 

•  No  measurable  effect 

•  Should  be  dropped  as  an  item  to 

consider 

{ 
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Turoff's  Policy  Delphi  Scales  (continued) 


CONFIDENCE  (IN  VALIDITY,  ARGU¬ 
MENT  OR  PREMISE) 

DESCRIPTIVE  GUIDELINES 

Certain 

•  Low  risk  of  being  wrong 

•  Decision  based  upon  this  will  not 
be  wrong  because  of  this  "fact" 

•  Most  inferences  drawn  from  this 
will  be  true 

Reliable 

•  Sonne  risk  of  being  wrong 

•  Willing  to  make  a  decision  based 
on  this  but  recognizing  some 
chance  of  error 

•  Some  incorrect  inferences  can  be 
drawn 

Risky 

•  Substantial  risk  of  being  wrong 

•  Not  willing  to  make  a  decision 
based  on  this  alone 

•  Many  incorrect  inferences  can  be 
drawn 

Unreliable 

•  Great  risk  of  being  wrong 

•  Of  no  use  as  a  decision  basis 
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Turoff's  Policy  Delphi  Scales  (continued) 


DESIRABILITY  (EFFECTIVENESS  OF 
BENEFITS) 

DESCRIPTIVE  GUIDELINES 

Very  desirable 

•  Will  have  a  positive  effect  and  little 
or  no  negative  effect 

•  Extremely  beneficial 

•  Justifiable  on  its  own  merit 

Desirable 

•  Will  have  a  positive  effect,  nega¬ 
tive  effects  minor 

•  Beneficial 

•  Justifiable  as  a  by-product  or  in 
conjunction  with  other  items 

Undesirable 

•  Will  have  a  negative  effect 

•  Harmful 

•  May  be  justified  only  as  a  by¬ 
product  of  a  very  desirable  item, 
not 

•  justified  as  a  by-product  of  a  de¬ 
sirable  item 

Very  undesirable 

•  Will  have  a  major  negative  effect 

•  Extremely  harmful 

•  Not  justifiable 

( 
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Turoff's  Policy  Delphi  Scales  (continued) 


PROBABILITY  (LIKELIHOOD) 

DESCRIPTIVE  GUIDELINES 

Very  probable 

•  Almost  certain  to  occur 

•  Strong  indication  of  this  happen¬ 
ing 

Probable 

•  Better  than  a  50-50  chance  of  oc¬ 
curring 

•  Some  indications  of  this  happen¬ 
ing 

Either  way 

•  About  50-50 

•  Could  go  either  way 

Improbable 

•  Less  than  a  50-50  chance  of  oc¬ 
curring 

•  Some  indications  of  this  not  hap¬ 
pening 

Very  improbable 

•  Almost  certain  not  to  occur 

•  Strong  indications  against  this 
happening 

> 
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Turoff's  Policy  Delphi  Scales  (continued) 


FEASIBILITY  (PRACTICALITY) 

DESCRIPTIVE  GUIDELINES 

Definitely  feasible 

•  No  hindrance  to  implementation 

•  No  R&D  required 

•  No  political  roadblocks 

•  Acceptable  to  the  public 

Possibly  feasible 

•  Some  indications  this  is 
implementable 

•  Some  R&D  required 

•  Further  consideration  or  prepara¬ 
tion  to  be  given  to  political  or 
public  reaction 

Definitely  infeasible 

•  All  indications  are  negative 

•  Unworkable 

•  Cannot  be  implemented 
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